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>  Background 

>  Acquisition  Challenges  and  Success 

>  The  Acquisition  Environment 

>  Contract  Requirements 

>  Key  Technical  Performance  Assessment  Areas 

□  Process 

□  Quality  Gates 

□  Software  Products 

□  Software  Testing 

>  Summary 
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Programs 

US  Air  Force  C-130AMP 

Software  Engineering  Advisory 
and  Assistance  Services 

-  9  years  (2001  -  Present) 

Lockheed  Martin  C-130J  Hercules 

Software  Subcontract 
Management 

-  4  years  (1995  -  1999) 


Roles 

Integrated  Product  Teams  Support 

Systems  Integration  Facility  (SIF) 
Operational  Flight  Program  (OFP)  Software 
Systems  Requirements,  Design  &  Test 

Supplier  Manager 
Review  and  approve  SDRL  items 
Monitor  supplier  activities 
Witness  acceptance  testing 
Coordinate  with  FAA  DER 


FAA  NAS  Plan  Programs 

Software  Engineering  Advisory 
and  Assistance  Services 

-  10  years  (1984 -1994) 


(AAS)  System  Development  Manager 

(TDWR’  SPO  Software  Lead 

Software  Subject  Matter  Expert  [e.g.,  VSCS 
MLS1,  RCE1,  NADIN  II,  MCCP/MCC2) 

1  Terminated  for  Default:  Deposed  by  AT&T  (RCE),  GAO  Audit  (MLS) 

2  Terminated  for  Convenience 


Plus  a  foundation  of  19-years  Software  Development  and  Process  Improvement 

United  States  Patents  #4451702,  #4479034 


The  Software-Intensive  Systems 

Crisis 
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How  many  of  you  heard-about  Software-Intensive 
Systems  that  have  experienced  cost  overruns, 
schedule  overruns,  and  performance  problems? 

□  Studies  shown  that  technical  performance,  cost, 
and  schedule  risks  are  inherent  for  Software- 
Intensive  Systems  [GAO  1999] 

■  GAO  1999  (U.S.  General  Accounting  Office,  “High  Risk  Series:  An  Update.  “GAO/HR-99-1 .  Washington, 

D.C.:GAO,  Jan.  1999.  ). 

□  70%  of  the  Pentagon’s  96  major  weapons  programs 
over  budget,  totaling  295  billion  [GAO  2009] 

■  GAO  2009  (U.S.  General  Accounting  Office,  “DEFENSE  ACQUISITIONS:  Charting  a  Course  for  Lasting  Reform”, 
GAO-09-663T,  Washington,  D.  C.  http://www.qao.gov/new.items/d09663t.pdf ) 

■  A  Solution:  Weapon  Systems  Acquisition  Reform  Act  of 
2009  (Public  Law  1 11-23,  22  May  2009) 

■  Ensure  that  the  problems  are  diagnosed  and  adjustment  are  made  in  the  process 
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Objectives 
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>  Illustrate  how  Software  Engineering  Advisory  and  Assistance 
Services  help  organizations  achieve 

□  Acquisition  excellence 

□  Objectives  of  the  Weapon  Systems  Acquisition  Reform  Act  of  2009  (PL 
1 1 1-23),  Title  I  Sec  103  Performance  Assessments  and  Root  Causes 
Analysis  for  Major  Defense  Acquisition  Programs 

□  Higher  software  development  capability  maturity 

>  Discuss  how  Key  Technical  Performance  Assessment  Areas 

□  Enables  determining  accuracy  and  adequacy  of  supplier’s  process  and 
deliverable  software  products 

□  Provides  measurable  results  for  determine  effectiveness  of  the 
supplier’s  process  and  quality  of  the  deliverable  software  products 

>  Provide  detailed  Practical  Examples/Recommendations  from  major 
defense  and  federal  acquisition  programs 


Acquisition  Challenges  and  Success 
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>Why  is  Software  Acquisition  a  Challenge? 

Examples 

□  Design  constraints  make  software  acquisition  and  development 
extremely  critical 

■  Application  domain  -  Operational  Flight  Program,  Air  Traffic  Control 

■  Software  size  -  Source  Lines-of-Code  (SLOC)  in  Millions 

■  Complexity  -  real-time  embedded  systems  of  systems 

■  Reliability  -  failure-free  software 

■  Safety-critical  -  RTCA/DO-178B 

□  Software  size  estimation  -  critical  factor  in  determining  cost, 
schedule,  and  effort  [Jones  2004]  [Jones  1999] 

■  Software  size  estimation  typically  driven  by  the  supplier’s 
agreement  terms  - 

■  Contract  vehicle  (Fixed-Price,  Cost-Reimbursement) 

■  Statement  of  work 

■  Deliverables  (Contract  Data  Requirements  List-CDRL) 

■  Technical  requirements  (safety-critical) 

■  Supplier’s  software  development  capability  maturity 


Acquisition  Challenges  and  Success 
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>Why  is  Software  Acquisition  a  Challenge? 

□  Acquires  must  take  overall  accountable  for  satisfying  the  user 
while  allowing  the  supplier  to  perform  the  tasks  to  delivery  the 
product 

□  Obstacles 

■  Inadequate  training  in  software  acquisition 
planning  and  execution 

■  Inadequate  resources  and  funding 

■  Capability  maturity  fails  to  match  the  supplier 

■  Inability  to  recognize  quality  work 

“Acquirers  must  recognize  quality  work  before  they  can  require  and  accept  it” 


— Watts  Humphrey,  2009 


Examples  of  Acquisition  Problems 
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FAA  NAS  Programs 

O  AAS  0  Inadequate  requirement  baseline  control 

o  Cost  and  Schedule  Overruns 
o  Restructured  in  1994 

-  contract  cost  increased  from  $3.6 
billion  to  $7.6  billion 

o  NADIN  II  o  Cost  and  Schedule  Overruns 

O  MCCP/MMC  0  Termination  for  Convenience 

O  M  LS  0  Termination  for  Default 

O  RCE  o  Termination  for  Default 


RCE:  Deposed  by  AT&T  BEFORE  THE  DEPARTMENT  OF  TRANSPRTATION 
BOARD  OF  CONTRACT  APPEALS  DOT  BCA  No.  2479 


Cornerstone  of  the 
FAA  NAS  Plan 


Examples  of  Acquisition  Success 
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FAA  NAS  Programs 

o  TDWR1  o  Delivered  First  Production  Unit  six  months  early 

o  Received  IEEE  Computer  Society  award 
o  Operational  at  45  Airports 

o  1991,  software  process  evaluated  a  SEI  CMM®  Level  3 

®  CMM  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 


acquirer  and  supplier  capability  maturity  levels  matched 


o  Production  completed 

o  100%  on-time  system  delivery  of  all  23  systems 

o  FAA  Contractor  of  the  Year  Award 
o  Human  Factors  Engineering  Society  Award 


acquirer  -  supplier  relationship  (communication) 


o  VSCS 
Upgrade 


1  Successful  Acquisition  of  FAA  Terminal  Doppler  Weather  Radar ,  Third  Annual  Conference  on  the 
Acquisition  of  Software-Intensive  Systems  (Experience  Track,  26  January  2004).  [Jones  2004-1] 

http  ://www.  sei.cmu.  edu/library/ abstracts/presentations/ionesi  an2004.  cfm 


Examples  of  Acquisition  Success 
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C^130"AMP1 

Boeing”  program  has  stayed  on  schedule  since  2005”  http://www.beurs.nl/nieuws/artikel.php?id=198658&taal=US 

o  Sep  19,  2006 -First  Flight  for  AMP1  (H2,  89-09101) 
o  Mar  25,  2007  -  First  Flight  for  AMP2  (H2.5,  91-01239) 
o  Sep  5,  2008  -  Completed  software  development  (Core  Complete  2.2) 

http://www.air-attack.com/news/article/3336/Boeinqs-C-130-AMP-Fliqht-Marks-Completion-of-Software-Development.html 

o  Jan  17,  2009  -  First  Flight  for  AMP3  (H3,  94-6704) 


Three  weeks  ahead  of  schedule 


o  Jan  28,  2010  -  DD-250  Software  Development  Workstation  to  Robins  AFB 

(SOF/EISE  Lab) 


Acquirer  -  Supplier  Relationship  (Communication) 


1  Source:  http://www.boeing.com/ids/news 
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TDWR  System  Software  Design 


CSSR  WBS  1.03.03 


Build  1  Build  2  Build  3  Build  A 


Cost/Schedule  Deviation 


RCE  System  Software  Design 


FQT  Progress 


SIF  CD/TK  Critical  Design  TIM 
3  -  4  Nov  2004 


(2)  (2)  (2)  (2)  (2)  (3)  (3)  (3) 


|ERCI  | 


Document  Review  Item 
Discrepancies 


TDWR  Software  Development  Documentation 


CDRL  Ite 


The  Acquisition  Environment 
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Program 

Management 


Software 
Acquisition 
Management 


Software 
Development 
Management 


Software 

Engineering 


Typical  Report  Items 

•  Issues  /  Status 

•  Milestone  Review  Readiness 

•  Discrepancies 

•  Corrective  Action  Request 


Software  CM 
Software  QA 


CMP 

SQPP 


Typical  Software 

Products 

•  Software  Development  Plan 
(SDP) 

•  Software  Configuration 
Management  Plan  (SCMP) 

•  Software  Quality  Assurance 
Program  Plan  (SQPP) 

•  Software  Requirements 
Specification  (SRS) 

•  Interface  Requirements 
Specification  (IRS) 

•  Software  Design  Description 
(SDD) 

•  Interface  Design  Description 
(IDD) 

•  Software  Test  Plan  (STP) 

•  Software  Test  Description 
(STD) 

•  Software  Test  Report  (STR) 

•  Software  Version  Description 
(SVD) 


SDP  -  Software  Development  Plan 
SEE  -  Software  Engineering  Environment 


CMP  -  Configuration  Management  Plan 
SQPP  -  Software  Quality  Program  Plan 
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Acquirer  /  Supplies  . . . 

Establish  agreed  upon  contractual  requirements 
Examples 

•  CDRL  Data  Item  Description  (Format  and  Content) 

•  Milestone  Review  (Entrance  and  Exit  Criteria) 

•  Mutual  understanding  of  the  functional  baseline 

Acquirer. . . 

Determines  accuracy  and  adequacy  of  supplier 
process  and  products 

•  Software  Development  Plan 

•  Software  Requirements  Specification 

Determines  product  readiness  for  Milestone  Reviews 

Supplier. . . 

Provides  tangible  evidence  of  product’s 
design,  status,  and  progress 


Practical  Examples  -  Acquirer 
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>  Software  engineering  should  be  integrated  in 

ALL  Software-Related  Source  Selection  Factors 

>  Software  acquisition  team  should  have  software 
expertise 

□  Application  domain,  acquisition,  process,  project 
management,  engineering,  and  safety,  as  needed 

>  The  software  acquisition  team  should  have 
adequate  resources  and  funding  to  perform  the 
acquisition  activities 

>  A  software  lead  should  be  designated  to  be 
responsible  for  establishment  and  managing  the 
software  acquisition  activities 


“Acquirers  must  recognize  quality  work  before  they  can  require  and  accept  it” 

— Watts  Humphrey 


Practical  Examples  -  Supplier 
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>  The  supplier  should  follow  a  set  of  defined  software  processes 
tailored  from  the  organization’s  standard  processes 

>  The  supplier  should  have  documented  and  institutionalized 
software  plans 

□  Software  Development  Plan  (SDP) 

□  Software  Configuration  Management  Plan  (SCMP) 

□  Software  Quality  Assurance  Plan  (SQP) 

>  The  supplier  software  plans  should  provide  the  acquirer  with 

□  Insight  into  the  processes,  procedures,  and  desk  instructions 

□  Tools  and  methods  used 

>  The  supplier  development  environment  should  be  augmented  by 
management  practices 

□  Measuring  and  monitoring  progress 

□  Judging  the  quality  of  the  software 

□  Validating  the  deliverable 

□  Conducting  milestone  reviews  and  in-process  reviews 


Contract  Requirements 
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>  Why  have  Contract  Data? 

□  Enables  the  acquirer  to  assess  the  accuracy  and  adequacy  of 
the  deliverable  software  product 

□  Enables  the  supplier  to  determines  production  methodology  and 
the  required  resources  for  the  product 


Contract  data  benefits  the  acquirer  and  supplier 


>  Key  Software-Related  Contract  Data  in  the  Request- 
For-Proposal 

□  Statement  of  Work  (SOW)/Statement  of  Objective  (SOO) 

□  Contract  Data  Requirements  List  (CDRL)  Items 

□  System  Specification 

□  Data  Rights 


Success  of  an  acquisition  is  directly  linked  to  the  quality  of  the  RFP 

—  (Army  2007) 


sow/soo 
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>  What  is  the  Statement  Of  Work  (SOW)  / 
Statement  Of  Objectives  (SOO)? 

□  Basis  for  communicating  acquirer  requirements  to  the  supplier 

■  SOW  defines  specific  tasks 

■  SOO  defines  objectives 

□  Primary  document  for  translating  management  requirements  into 
contractual  tasks  /  objectives 

□  Sufficient  detail  should  be  provided  to  allow  the  supplier  to 
scope  the  effort,  cost  it,  and  provide  a  responsive  technical 
solution 

□  'asking  information  should  be  defined  for  the  preparation  of 
deliverable  data  (artifact) 

■  Each  tasking  statement  should  reference  applicable  Contract  Data 
Requirements  List  (CDRL)  item  which  will  be  delivered  by  that 
task. 


The  SOW/SOO  should  not  tell  the  supplier  how  to  do  the  required  work 
The  SOW/SOO  should  not  specify  selection  of  major  software  components 


Contract  Data  Requirements  List 

CDRL 
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>  What  is  the  Contract  Data  Requirements  List 
(CDRL)  Items? 

□  A  natural  by-product  of  the  development  process  to  capture 
results  of  each  activity 

□  Primary  vehicle  for  acquiring  software  data 

□  A  list  of  authorized  data  requirements  for  a  specific  procurement 
that  forms  a  part  of  the  contract. 

□  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS) 

Subpart  215.470  Estimated  Data  Prices 

■  Requires  a  CDRL  (DD  Form  1423)  when  delivery  of  data  is  required 

□  CDR1  :ems  referenced  in  the  Statement  of  Work  (SOW) 
describing  the  development  effort 


CDRL  Items  absolutely  essential  for  managing  software  development 


Practical  Examples 
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>  Contractual  Requirements  should  be  agreed  upon 

□  CDRL  DID  Format  and  Content 

>  Software  CDRL  items  should  be  delivered  prior  to  the 
milestone  reviews  to  allow  significant  time  to  enable: 

□  Acquirer  evaluation  and  to  provide  discrepancies 

□  Supplier  to  disposition  the  discrepancies 

>  Miiestone  reviews  should  include  agreed  upon  supplier 
discrepancies  disposition 

>  software  CDRL  items  should  be  prepared  by  the 
software  team 

□  Reviewed  by  all  applicable  distribution  addressee  organization 

□  Approved  by  either  the  appropriate  Chief  Engineer,  Program 
Manager  or  Data  Requirements  Review  Board 


System  Specification 
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>  What  is  the  System  Specification? 

□  Establish  top-level  technical  performance,  design,  development, 
integration,  and  verification  requirements 

□  Sound  system  requirements  are  the  backbone  of  good  Key 

Performance  Parameters  -  essential  to  the  development  of 
effective  capabilities 

□  Examples  of  requirement  statements 

■  All  safety-critical  software  shall  be  modified  or  developed  in 
accordance  with  the  requirements  of  RTCA/DO-178B  or  equivalent 
level  of  safety. 

■  All  newly  developed  software  shall  be  written  in  a  higher  order 
language  (HOL). 

■  Meteorological  algorithms  shall  be  implemented  in  high  order 
language  (HOL) 

•  Use  of  commercial  software  shall  be  approved  by  the  SPO 


Data  Rights 
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>  What  are  Data  Rights? 

□  Enable  the  use,  maintenance,  and  replication  of  the  software  data 

>  Data  Rights  Categories 

.  Unlimited  rights  -  right  to  use,  modify,  reproduce,  release,  in  whole  or 
in  part,  in  any  manner  and  for  any  purpose  whatsoever,  and  to  have  or 
authorize  others  to  do  so.  Associated  with  computer  software  developed 
exclusively  with  acquirer  funds. 

□  Acquirer  Purpose  rights  -  rights  to  use,  modify,  reproduce,  release, 
within  the  acquirer’s  organization/company  without  restriction.  Software 
development  with  mixed  acquirer  and  supplier  funding. 

□  Restricted  data  rights  apply  only  to  noncommercial  computer  software 
and  mean  that  the  acquirer’s  rights  are  as  set  forth  in  a  Restricted 
Rights  Notice.  Supplier  funds  all  development. 

Secretary  of  the  Air  Force  Memo  -  Data  Rights  and  Acquisition  Strategy  (3  May  06)  - 


directing  the  acquisition  of  technical  data  and  associated  rights  to  be  addressed 
specifically  in  all  Acquisition  Strategy  Plans,  reviews,  and  associated  planning 
documents  for  Acquisition  Categories  (AC AT)  programs  -  software  intensive  systems 
and  subsequent  source  selections. 
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Technical  Performance  Assessment 
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Supplier 

•  Process  Definition 

•  Processes 

•  Deliverable  Software  Products 


T 


Perform  Technical  Performance 

Contract 

Assessment 

Reports 

•  SOW 

•  Process 

•  Corrective  Action  Report 

•  CDRL 

•  Review  Discrepancies 

•  Sys  Spec 

•  Quality  Gates 

•  Performance  Measures 

•  Data  Rights 

t? 

•  Software  Products 

_ _ _ r 

•  Software  Testing 

Software  Acquisition  Team 


•  SW  Lead 

•  SW  Engineering,  SW  Quality,  SW  Configuration,  SW  Safety 


Gaining  Visibility  to  Reduce  Technical  Performance  Risks 


Technical  Performance  Assessment 
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Technical  Performance  Assessment 

-  An  evaluation  technique  in  which  process,  quality  gates, 

software  product,  and  software  testing  are  examined  in 

detail  to  detect  defaults  and  compliance  with  standards 

>  Objectives 

□  To  reduce  software  development  technical  performance  inherent 
risks 

>  Benefits 

□  Ensures  compliance  with  contractual  requirements  and 
supplier’s  defined  software  process  and  plans 

□  Provides  visibil  into  accuracy  and  adequacy  of  the  of  the 
deliverable  products 

□  Provides  measurable  results  for  determine  effectiveness  of  the 
process  and  the  quality  of  the  products 

□  Provides  feedback  to  improve  the  software  process 

□  deduce  software  development  technical  performance  risks 


Technical  Performance  Assessment 
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>Key  Technical  Performance  Assessment 
Areas 

□  Process 

□  Quality  Gates 

□  Software  Products 

□  Software  Testing 


Meeting  the  objectives  of  the  DoD  Weapon  Systems  Acquisition  Reform  Act  2009 
Public  Law  111-23-May  22,  2009 


Process  Area 
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>  Objective 

□  Ensures  compliance  with  contractual  requirements 
and  supplier’s  defined  software  process  and  plans 

■  software  management,  engineering,  configuration 
management,  and  quality  assurance  activities 

Process  Assessment  key  focus  is  “what  is  done  and  the  product  being  built” 

>  Examples  of  Software  Plans 

□  Software  Development  Plan  (SDP) 

□  Software  Configuration  Management  Plan  (SCMP) 

□  Software  Quality  Assurance  Plan  (SQAP) 


The  Contract  must  provide  mechanism  to  gain  access  to  supplier  process  and  plans 
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>  Contract:  SOW 

□  Maintain  the  software  development  process,  and  SDP 

□  Process  remain  up  to  date  with  the  current 
development  activities,  and  the  SDP  remains 
consistent  with  the  actual  activities  being  performed. 

□  Implementation  of  software  configuration 
management  in  accordance  with  the  approved 
configuration  management  and  software  development 
plans  using  IEEE/EIA  12207.0,  12207.1  and  12207.2 
as  guides 

□  Development  and  maintenance  of  a  SDP  for  each 
supplier  furnished  avionics  Operational  Flight 
Program  (OFP)  Computer  Software  Configuration 
Item  (CSCI) 
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Practical  Examples 
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> Activities  Performed 

□  Acquirer  reviews  supplier’s  defined  software  process 

□  Acquirer  evaluates  supplier’s  plans  against  SOW 

■  If  CDRL  items,  provides  discrepancies 

□  Acquirer  monitors  supplier  management  and 
engineering  activities  in  accordance  with  supplier’s 
defined  software  process  and  plans 

□  Acquirer  conducts  periodic  reviews  and/or  audits  of 
the  supplier’s  software  configuration  management 
and  software  quality  assurance  activities 

□  Acquirer  provides  discrepancies  and/or  audit  reports 
to  the  supplier  and  acquirer  management 


Quality  Gates  Area 
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>  Objective 

□  Determines  the  adequacy  of  the  supplier’s  efforts  in  performing 
the  software  development  activities,  surface  and  resolve  issues, 
and  provide  feedback 

>  Examples  of  Quality  Gates 

■  Program  Management  Review  (PMR) 

■  Milestone  Reviews 

■  Technical  Interchange  Meetings  (TIM) 

□  Typical  Milestone  Reviews:  Now:  No  official  standards 

■  Software  Specification  Review  (SSR) 

■  Preliminary  Design  Review  (PDR) 

■  Critical  Design  Review  (CDR) 

■  Test  Readiness  Review  (TRR) 

>  Example  of  Audits 

■  Functional  /  Physical  Configuration  Audits 


Practical  Examples 


Support  Systems  Associates,  Inc. 


800  Park  Drive 


Warner  Robins,  GA  31088 


>  Contract:  SOW  (Milestone  Reviews) 

□  Conduct  Milestone  Reviews  in  accordance  with  the 
Contractor  Integrated  Master  Plan  (IMP). 

□  Example  of  an  IMP  Attribute 

■  Event:  Conduct  Critical  Design  Review  (CDR) 

■  Accomplishments:  Software  Detailed  Design  performed  and 
documented  in  the  Software  Detailed  Design  document 
(CDRL  Item) 

■  Criteria: 

■  Peer  reviewed  and  placed  under  Configuration  Management 
Control 

■  CDRL  Item  submitted  for  Acquirer  review 


Milestone  Reviews  should  be  evidence-based 


Critical  Success  Factors  for  Milestone  Review  Risk  Identification  Dr.  Barry  Boehm 


Practical  Examples 
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>  Activities  Performed  (Milestone  Reviews) 

□  Supplier  prepares  the  documents  in  accordance  with  the  CDRL 
Item  DID  and  places  under  configuration  control 

□  Supplier  delivers  CDRL  Items  in  accordance  with  the  CDRL 
Items 

□  Acquirer  evaluates  the  CDRL  Items  to  determine  evidence- 
based  (acceptability)  and  provides  discrepancies  to  supplier 

■  Maturity  of  CDRL  items  serves  as  entrance  criteria  for  milestone 
review 

□  Supplier  disposition  CDRL  items  discrepancies  prior  to  the 
Milestone  Reviews 

□  Supplier  conducts  the  Milestone  Reviews  in  accordance  with 
supplier’s  process 

□  Acquirer  and  supplier  agree  upon  supplier’s  CDRL  Item 
discrepancies  disposition 


Key  Focus — What  is  done  and  the  product  being  built 


Software  Products  Area 
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>  Objective 

□  Provides  insight  into  the  accuracy  and  adequacy  of 
the  supplier’s  deliverable  software  products 

□  Provides  measurable  results  for  determining  the 
quality  of  the  software  products  and  process 
effectiveness 

□  Ensures  software  products  meets  contractual 
requirements 

■  Statement  of  Work  (SOW) 

■  Contract  Data  Requirements  List  (CDRL)  Item  -  Data  Item 
Description  (DID) 

□  Provides  evidence-based  for  milestone  reviews 
(quality  gates)  to  determine  readiness 


Practical  Example 
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>  Activities  Performed 

□  After  contract  award,  acquirer  and  supplier  agree 
upon  the  CDRL  DID  format  and  content 

□  Prior  to  delivery  to  the  acquirer,  supplier  evaluates 
CDRL  items  and  places  under  configuration  control 

□  Supplier  delivers  CDRL  items  prior  to  the  milestone 
review  to  allow  significant  time  for  acquirer  detailed 
evaluation  and  disposition  of  discrepancies 

■  CDRL  delivery  and  discrepancies  disposition  serves  as 
entrance  criteria  for  the  milestone  review 

□  Acquirer  provides  discrepancies  and 
recommendations  to  supplier  within  an  agreed  upon 
time  after  receipt  of  the  CDRL  items 

□  Supplier  incorporates  agreed  upon  corrections 


Practical  Example 


K9I 

Support  Systems  Associates,  Inc.  800  Park  Drive  Warner  Robins,  GA  31088 

>  Example  of  Document  Review  (DR)  Form 

□  Capture  CDRL  items  evaluation  discrepancies 

■  Documentation  Identification  (e.g.,  CDRL  Title,  CDRL  Item 
Number,  Document  Title,  Document  Number/Date/Revision) 

■  Comment  Location  (e.g.,  Page,  Paragraph,  Figure) 

■  Comment  (e.g.,  consistency,  completeness,  correctness, 
ambiguous,  traceability,  etc) 

■  Recommendation  /  Suggestion 

□  Capture  supplier  disposition  of  discrepancies 

■  Concur  /  non-concur  /  Action 

□  Capture  acquirer  agreement 

□  Track  Closure 

■  Date  /  Action 


Practical  Example 
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>  Examples  of  CDRL  Items  Assessment  Criteria 

□  Compliance  with  DID  format  and  content 

□  Completeness  (e.g.,  missing  requirements,  testing, 
interfaces,  etc.) 

□  Traceability  (e.g.,  test  traced  to  requirements,  etc.) 

□  Consistency  with  upper  level  documents 

□  Internal  consistency 

□  Ambiguity  of  requirements  (understandable, 
testable?) 

□  Conflicting  requirements 

□  Test  coverage  of  requirements 

□  Appropriate  analysis,  design,  and  coding  techniques 
used 
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Practical  Example 
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>  Bidirectional  Traceability 

□  Required  by  the  )DRL 
Item  DID 

□  Allocation  ensures  the  right 
products  been  built 

□  raceab  ty  ensures  the 
evolving  product  is  not 
expanding  the  scope 

□  Reduce  effort  required  to 
determine  change  impact 

□  Should  be  documented  in 
a  requirements  database 

■  Examples:  DOORS®, 

RTM 


©DOORS  is  a  trademark  of  Telelogic  AB 
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^  C-130  AMP  Contract 

□  8  Software-Related  CDRL  Items  specified  by  the  SOW 

■  SRS  (A012),  IRS  (A013),  SDD  (A014),  IDD  (A015),  STP  (A016),  STD 
(A017),  STR  (A018),  SPS  (A019) 

■  Final  submittal  60  days  before  EMD  completion  for  the 

SIF  nodes  and  final  submittal  60  days  after  software  FCA  for 
other  CSCIs. 

■  The  CDRL  noted  :  “Only  final  version  of  data/document  to  be 
formally  delivered  in  accordance  with  the  above  stated  milestone.  Any 
initial,  preliminary,  draft,  or  other  interim  versions  of  the 
data/document  referenced  in  the  contractor’s  IMP  will  be  made 

_ available  informally  to  the  government” _ 

>  C-130  AMP  SPO  Activities 

□  Software  IPT  responsible  for  MP  OFP  Software  CSCIs 

□  SIF  IPT  responsible  for  SIF  Hardware  and  Simulation  Software 
CSCIs 

□  Submitted  over  3500  Document  Comment  Items  (DCI) 

■  90%  acceptance 


Practical  Examples 
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>  FAA  NAS  (TDWR)  Contract 

□  16  CDRL  Items  specified  by  the  SOW 

□  Submittal  (preliminary  and  final)  linked  to  design 
review  (e.g.,  SSR,  PDR,  etc) 

□  Acquirer  approval  within  30-calendar  days 

>  Raytheon 

□  45  Total  CDRL  Items  delivered 

>  TDWR  Software  IPT 

□  Over  4300  Review  Items  Discrepancies  (RID) 
approved 


Software  Testing  Area 
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>  What  is  Software  Testing? 

□  Software  development  involves  a  series  of  activities  in 
which  opportunities  for  human  induced  defects  are 
enormous 

■  46%  -  60%  of  all  software  defects  originate  in  the  software 
requirements  analysis  phase  [Endves  1975]  [Voges  1979] 

□  Software  Testing  is  the  quality  assurance  technique 
used  to  evaluate  the  “ as-built ’  software  product  to 

ensure  the  probability  of  failure  due  to  latent  defects 
is  low  enough  for  acceptance 

□  Software  testing  consists  of  three  levels  of  testing 

■  Unit  Testing,  Integration,  and  Formal  Qualification  Testing 


Software  testing  represents  the  ultimate  evaluation  of  the  software  requirements,  design,  and 
coding  activities  [Jones  1993-1] 


Software  testing  can  make  the  software  product  more  reliable  and  usable  [Musa  1987]  [Dunnl984] 


Software  Testing  Area 
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>  What  is  required  in  the  Contract? 

□  Unit  Testing,  Integration,  and  Formal  Qualification  Testing  (FQT) 
activities  and  artifacts  should  be  documented  in  the  supplier’s 
defined  software  process  and  the  Software  Development  Plan 

□  FQT  activities  and  artifacts  should  be  specified  in  the  Contract 
SOW 

Examples 

□  Software  Test  Plan  (STP)  -  DI-IPSC-81438A 

■  Describes  plans  for  qualification  testing,  test  environment, 
identify  tests  to  be  performed,  and  schedule 

■  Tests  trace  to  requirements  in  the  SRS 

□  Software  Test  Description  (STD)  -DI-IPSC-81439A 

•  Describes  the  test  preparation,  test  cases,  and  test 
procedures 

□  Software  Test  Results  (STR)  -  DI-IPSC-81440A 

•  A  record  of  the  qualification  testing 
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>  Activities  Performed 

□  Supplier  delivers  software  test  CDRL  Items  at  designated  quality 
gates  -  Milestone  Reviews  (i.e.,  PDR,  CDR,  and  TRR)  in 
accordance  with  CDRL  Items 

□  Supplier  conducts  Test  Readiness  Review  (TRR)  in  accordance 
with  supplier’s  process 

□  Supplier  conducts  software  Formal  Qualification  Testing  (FQT)  in 

accordance  with  supplier’s  process,  software  test  plan,  and 
software  test  procedures 

□  Acquirer  and  supplier’s  Software  Quality  Assurance  witness  all 
software  FQT  activities 

□  During  FQT  activities,  supplier  documents  and  disposition  test 
anomalies  detected  in  accordance  with  supplier  process 

□  Supplier  performs  corrective  action,  as  required,  in  accordance 
with  Supplier’s  process 
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>  Example  of  Supplier  Corrective  Action  Process 

□  Establish  and  maintain  a  Configuration  Change 
Board 

■  Determine  the  severity  of  all  problems  detected  or  changes 
requested 

■  Analyze  the  changes  to  determine  impact  to  the  work 
product,  related  work  product,  and  schedule 

■  Analyze  the  problem  closure  to  determine  the  impact  to  the 
software  release  milestone 

□  Perform  corrective  action  life  cycle  in  accordance  with 
Supplier’s  process 


Change  control  system  should  be  used  to  determine  the  aspects  of 

and  of  previous  activities 


Practical  Example 
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Typical  Corrective  Action  Life  Cycle 


Software  Testing  Area 
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>  How  much  testing  is  enough? 

□  Complete  test  coverage  is  generally  not  possible  [Jones  1993-1] 

□  Test  Case  design  methodology  should  be  documented 

□  Acquirer  and  supplier  should  mutually  agree  on  completion 
criteria 

Examples 

■  Completion  of  a  number  of  test  runs  with  no  open  priority  1  and  2 
severity  problems 

□  Acquirer  and  supplier  should  establish  a  failure  intensive 
objective  (FIO)  using  a  software  reliability  growth  model: 
Examples 

■  Time-Between-Failure  Models 

■  Error-Count  Model 


Acquirer  and  supplier  face  a  difficult  decision  when  to  release  the  software  product 
Complete  test  coverage  is  generally  not  possible. .  .[Jones  1993-1] 
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Achieving  Acquisition  Excellence  -  Changing  The  Game 

>  Software  Engineering  Advisory  and  Assistance  Services 

□  Effective  software  acquisition  expertise  is  essential 

>  Supplier 

□  Performs  effective  software  estimation  ( software  size,  cost,  and 
schedule)  early  and  during  the  life  cycle 

□  Mature  software  development  capability 

>  Technical  Performance  Assessment 

□  Determine  accuracy  and  adequacy  of  supplier’s  process  and 
deliverable  software  products 

□  Ensure  “as-built”  meets  software  requirements 

□  Provide  measurable  results  for  determining  effectiveness  of  the 
supplier’s  process  and  quality  of  the  deliverable  software 
products 
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Summary 
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>  Way  Forward  -  Changing  The  Game 

□  Establish  software  acquisition  team  with  knowledge 
and  skills 

■  Effective  software  engineering  advisory  and  assistance  services 

□  Establish  adequate  resources  and  funding 

□  Getting  things  right  from  the  start 

■  Establishing  effective  and  efficient  software  Contractual 
Requirements 

■  Statement  of  Work,  Contract  Data  Requirement  List  Items, 
Systems  Specification,  Data  Rights 

■  Validating  software  estimates  (Size,  Cost,  and  Schedule) 

□  Gain  visibility  into  technical  performance  risks 


Establishing  more  discipline  and  accountability 


Summary 
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>  Improvements  in  Software  Acquisition 


□  Public  Law  107-314  Section  804  of  the 
National  Defense  Authorization  Act,  released 
in  December  2002  [Section  804-2003] 


□  The  best  practice  model  Capability  Maturity 
Model®  Integration  (CMMI®)  for  Acquisition 


®  Capability  Maturity  Model,  CMM,  CMM  Integration,  and  CMMI 
Registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 


Questions  ? 
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What  are  m  doing  to  achieve  acquisition  e ace  lie  nee? 


James  E.  Jones 

Support  Systems  Associates,  Inc. 
Warner  Robins,  GA  31088 
Email:  iiones@ssai.org 
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AAS 

Advanced  Automated  System 

FQT 

Formal  Qualification  Testing 

ACAT 

Acquisition  Category 

IDD 

Interface  Design  Description 

AMP 

Avionics  Modernization  Program 

IRS 

Interface  Requirements  Specification 

ATC 

Air  Traffic  Control 

MP 

Mission  Processor 

CDR 

Critical  Design  Review 

NAS 

National  Airspace  System 

CDRL 

Contract  Data  Requirements  List 

OFP 

Operational  Flight  Program 

CIP 

Capital  Investment  Plan 

OFP 

Operational  Flight  Program 

C  NS/ATM 

Communications/Navigation  Surveillance  /  Air  Traffic 

PCO 

Procuring  Contracting  Officer 

Management 

PDR 

Preliminary  Design  Review 

CO 

Contracting  Officer 

SCM 

Software  Configuration  Management 

COTS 

Commercial  Off-The-Shelf 

SDD 

Software  Design  Description 

CPAF 

Cost-Plus  Award  Fee 

SOF 

Special  Operations  Forces 

CSCI 

Computer  Software  Configuration  Item 

SOO 

Statement  of  Objective 

CY 

Calendar  Year 

SOW 

Statement  of  Work 

DCI 

Document  Comment  Item 

SPO 

System  Program  Office 

DER 

Designated  Engineering  Representative 

SQA 

Software  Quality  Assurance 

DFARS 

Defense  Federal  Acquisition  Regulation  Supplement 

SRS 

Software  Requirements  Specification 

DID 

Data  Item  Description 

SSR 

Software  Specification  Review 

DoD 

Department  of  Defense 

STD 

Software  Test  Description 

DOORS 

Dynamic  Object-Oriented  Requirements  Systems 

STP 

Software  Test  Plan 

ECP 

Engineering  Change  Proposal 

STR 

Software  Test  Report 

EMD 

Engineering,  Manufacturing  and  Development 

SVD 

Software  Version  Description 

FAA 

Federal  Aviation  Administration 

TRR 

Test  Readiness  Review 

FFP 

Firm  Fixed-Price 

FFPI 

Firm  Fixed-Price  Incentive 
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Back-Up 


Typical  Supplier’s  Software  Process 

Definition 


Support  Systems  Associates,  Inc. 


800  Park  Drive 


Warner  Robins,  GA  31088 


r 

\ 

Activities 

V 

J 

_ ± 

_ 1 _ 

_ 1 _ 

Project  Results  and  Software  Work  Products 

sow/soo 
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>  Examples  of  Key  Software  Tasking 

□  Software  development  process  —  perform  software  management 
and  engineering  in  accordance  with  project’s  processes,  procedures, 
and  desk  instruction 

□  Software  management  —  manage  software  development  in 
accordance  with  software  plans  (e.g.,  SDP,  CMP,  SQPP) 

□  Software  engineering  -  perform  software  activities  (i.e., 
requirements  analysis,  preliminary  design,  detailed  design,  code  and 
unit  test,  integration,  and  formal  qualification  testing) 

□  Software  tools  and  environment  -  used  to  produce  the  software 

□  ?/s/c  management  —  established  and  maintained  risk  management 
systems 

□  Milestone  reviews  -  conduct  milestone  reviews  with  the  acquirer 
Software  Specification  Review  (SSR),  Preliminary  Design  Review 
(PDR),  Critical  Design  Review  (CDR),  and  Test  Readiness  Review 
(TRR)  [used  as  quality  gates] 

□  Technical  Interchange  Meetings  -  used  to  identify  and  resolve 
technical  issues 

□  In  Process  Reviews  -  determine  evidence  readiness 


Examples  of  SOW  Software 
Enaineerina  Taskin 
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>  Software  Requirements  Analysis 

□  Perform  software  requirements  analysis.  Prepare  software 
requirements  specification  document  in  accordance  with  CDRL  No. 
A001 

>  Software  Design 

□  Perform  software  preliminary  and  detailed  design.  Prepare  software 
design  description  document  in  accordance  with  CDRL  No.  A002 

>  Software  Test  Planning 

□  Perform  software  test  planning.  Prepare  software  test  planning 
document  in  accordance  with  CDRL  No.  A003 

>  Software  Test  Description 

□  Define  the  test  environment,  test  cases  and  test  procedures.  Prepare 
software  test  description  document  in  accordance  with  CDRL  No.  A004 

>  Software  Formal  Qualification  Testing 

□  Conduct  software  formal  qualification  in  accordance  with  the  test 
procedures  in  the  software  test  description  CDRL  A004.  Prepare  the 
test  report  document  in  accordance  with  CDRL  No.  A005 


Typical  Software  CDRL  Items 
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□  SOFTWARE  REQUIREMENTS  SPECIFICATION  (SRS)  - 

■  Describes  the  behavior  of  the  software  to  be  developed  and  methods  to  be  used  to 
ensure  each  requirement  has  been  met 

■  Basis  for  the  design  and  qualification 

■  Enables  the  acquirer  to  access  the  adequacy  of  the  software  requirements 

-  Interface  Requirements  Specification  (IRS)  -  DI-IPSC-81434A  may  be  appendix  to 
SRS 

□  SOFTWARE  DESIGN  DESCRIPTION  (SDD)  -  DI-IPSC-81435A 

■  Describes  the  design  needed  to  implement  the  software 

■  Interface  Design  Description  (IDD)-DI-IPSC-81436A,  may  be  appendix  to  SDD 

"  Database  Design  Description  (DBDD)-DI-IPSC-81437A,  may  be  appendix  to  SDD 

■  Describes  the  data  base  design  and  elements  (content  and  format) 

□  Software  Test  Plan  (STP)  -  DI-IPSC-81438A 

■  Describes  plans  for  qualification  testing,  test  environment,  identify  tests  to  be 
performed,  and  schedule 

□  Software  Test  Description  (STD)  -DI-IPSC-81439A 

■  Describes  the  test  preparation,  test  cases,  and  test  procedures  to  be  used  to  perform 
the  qualification  testing 

■  Enables  the  acquirer  to  access  the  adequacy  of  the  qualification  testing 

□  Software  Test  Results  (STR)  -  DI-IPSC-81440A 

■  A  record  of  the  qualification  testing 

■  Enables  the  acquirer  to  access  the  testing  and  its  results 

□  Software  Version  Description  (SVD)  -  DI-IPSC-81442A 

■  Identifies  and  describes  a  software  version  (“as-built”  software) 


CDRL  Item  Key  Blocks 
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>CDRL  Item  Key  Blocks  Elements 


Block 

Description 

4 

Authority  (Data  acquisition  Documentation  No.) 

Data  Item  Description  (DID1)  -  Defines  format  and  content  preparation 
instructions  for  data  product  generated  by  task  requirements 

Assist-Quick  Search  used  to  access  the  current  DID 

1  Should  be  tailored  to  meet  contract  requirements  (Block  16) 

5 

Contract  Reference  -  Reference  Statement  of  Work  paragraphs 

6 

Requiring  Office  -  Organization  have  primary  responsibility  for  reviewing 
the  data  and  recommending  acceptance/rejection  of  the  data 

8 

Approval  Code  -  (A)  Approved  by  the  Contracting  Officer 

Should  specify  approval  at  each  milestone  (e.g.,  SSR,  PDR,  CDR,  etc.) 

10,  11, 
12,  13 

Delivery  Requirements 

Should  be  associated  with  milestone  reviews  (e.g.,  SSR,  PDR,  CDR,  etc.) 

Weapon  Systems  Acquisition  Reform  Act 
of  2009  (Public  Law  111-23,  22  May  2009) 


Support  Systems  Associates,  Inc. 


800  Park  Drive 


Warner  Robins,  GA  31088 


TITLE  I— ACQUISITION  ORGANIZATION 

Sec.  101.  Cost  assessment  and  program  evaluation. 

Sec.  102.  Directors  of  Developmental  Test  and  Evaluation  and  Systems  Engineering. 

Sec.  103.  Performance  assessments  and  root  cause  analyses  for  major  defense  acquisition  programs. 

Sec.  104.  Assessment  of  technological  maturity  of  critical  technologies  of  major  defense  acquisition  programs  by  the  Director  of  Defense 
Research  and  Engineering. 

Sec.  105.  Role  of  the  commanders  of  the  combatant  commands  in  identifying  joint  military  requirements. 

TITLE  II— A  CQ  UISITION  POLICY 

Sec.  201.  Consideration  of  trade-offs  among  cost,  schedule,  and  performance  objectives  in  Department  of  Defense  acquisition  programs. 
Sec.  202.  Acquisition  strategies  to  ensure  competition  throughout  the  lifecycle  of 
major  defense  acquisition  programs. 

Sec.  203.  Prototyping  requirements  for  major  defense  acquisition  programs. 

Sec.  204.  Actions  to  identify  and  address  systemic  problems  in  major  defense  acquisition  programs  prior  to  Milestone  B  approval. 

Sec.  205.  Additional  requirements  for  certain  major  defense  acquisition  programs. 

Sec.  206.  Critical  cost  growth  in  major  defense  acquisition  programs. 

Sec.  207.  Organizational  conflicts  of  interest  in  major  defense  acquisition  programs. 

TITLE  III— ADDITIONAL  A  CQUISITION  PRO  VISIONS 

Sec.  301.  Awards  for  Department  of  Defense  personnel  for  excellence  in  the  acquisition  of  products  and  services. 

Sec.  302.  Earned  value  management. 

Sec.  303.  Expansion  of  national  security  objectives  of  the  national  technology  and  industrial  base. 

Sec.  304.  Comptroller  General  of  the  United  States  reports  on  costs  and  financial  information  regarding  major  defense  acquisition 
programs. 


